Papers with fine-grained assessment
CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models (2025.coling-main)
Copied to clipboard
Zhongzhi Li, Ming-Liang Zhang, Pei-Jie Wang, Jian Xu, Rui-Song Zhang, Yin Fei, Zhi-Long Ji, Jin-Feng Bai, Zhen-Ru Pan, Jiaxin Zhang, Cheng-Lin Liu
| Challenge: | Large language models excel in various language tasks, while large multimodal models effectively handle visual-language problems. |
| Approach: | They propose to use a multimodal multimodal model evaluation benchmark to evaluate model performance in Chinese K12 classrooms. |
| Outcome: | The proposed model evaluation tool is integrated with the CMMaTH dataset. |
PLAWBENCH: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice (2026.acl-long)
Copied to clipboard
Yuzhen Shi, Huanghai Liu, Yiran HU, Song Gaojie, Xu Xinran, Yubo Ma, Tianyi Tang, Li Zhang, Qingjing Chen, Feng Di, Wenbo Lv, Weiheng Wu, Kexin Yang, Sen Yang, Wei Wang, Rongyao Shi, Qiu Yuanyang, Yuemeng Qi, Zhang Jingwen, Sui Xiaoyu, Yifan Chen, Zhang Yi, An Yang, Bowen Yu, Dayiheng Liu, Junyang Lin, Weixing Shen, Bing Zhao, Charles L. A. Clarke, HU Wei
| Challenge: | Existing benchmarks for large language models (LLMs) are coarse, single-dimensional metrics and do not explicitly assess fine-grained legal reasoning. |
| Approach: | They propose a Practical Law Benchmark to evaluate large language models in real-world legal practice scenarios. |
| Outcome: | The proposed model is based on 850 questions and 13 scenarios with expert-designed evaluation rubrics. |
Reward Modeling for Scientific Writing Evaluation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing models for scientific writing evaluation are primarily optimized for general-purpose benchmarks with fixed scoring rubrics and evaluation criteria. |
| Approach: | They propose to train scientific writing evaluation models that leverage domain knowledge . they use a two-stage evaluation framework that optimizes evaluation preferences and refines reasoning capabilities . |
| Outcome: | The proposed model generalizes effectively across tasks and to previously unseen settings. |